1. Field of the Invention
The present invention relates to a system in which a plurality of CPUs and peripheral devices are distributedly connected to a network to share the peripheral devices by the CPUs and, more particularly, to a system in which a plurality of CPUs and peripheral devices are connected through a PCI Express switch connected through a network.
2. Description of the Related Art
As a specification of a bus for connecting a central processing unit (CPU) to peripheral devices such as a storage device, a network interface (NIC), and the like, a peripheral component interconnect (PCI) is widely prevalent. As a next-generation specification for the PCI, a PCI Express which serializes a parallel bus of the PCI to perform communication by a packet system with switching is standardized. An example of a PCI Express switch network formed by the PCI Express is disclosed in PCI Express Base Specification Revision 1.1, PCI-SIG, Mar. 28, 2005, pp. 30.
Referring to FIG. 1, a PCI Express switch network described in PCI Express Base Specification Revision 1.1, PCI-SIG, Mar. 28, 2005, pp. 30 includes a CPU 101, a route complex 102 realized by a chipset, a memory 103, a PCI Express switch 801, and a peripheral device 109.
The CPU 101 and the route complex 102 are connected to each other and the route complex 102 and the memory 103 are connected to each other by a high-speed communication system different from the PCI Express. On the other hand, the route complex 102 and the PCI Express switch 801 are connected to each other by a PCI Express bus, and the PCI Express switch 801 and the peripheral device 109 are connected to each other by a PCI Express bus. These components communicate with each other by a communication system conforming to the PCI Express.
The route complex 102 receives an instruction from the CPU 101, performs transferring of peer-to-peer communication between the CPU 101 and the peripheral device 109 and peer-to-peer communication between the memory 103 and the peripheral device 109. At this time, communication is performed between the route complex 102 and the peripheral device 109 by using a packet (TLP: Transaction Layer Packet) of the PCI Express. Therefore, a PCI Express switch network forms a hierarchical network in which the route complex 102 and the peripheral device 109 are used as a tree-structure route and a leaf, respectively. In this sense, in the PCI Express switch network, the route complex 102 side is called an upstream side, and the peripheral device 109 side is called a down stream side.
The PCI Express switch 801 transfers TLPs received from respective ports of the switch to ports of the PCI Express switch 801 to which the destination route complex 102 and the peripheral device 109 are connected. An example of the configuration of the PCI Express switch 801 is described in PCI Express Base Specification Revision 1.1, PCI-SIG, Mar. 28, 2005, pp. 34.
Referring to FIG. 2, the PCI Express switch 801 described in PCI Express Base Specification Revision 1.1, PCI-SIG, Mar. 28, 2005, pp. 34 includes an upstream PCI-PCI bridge 1101 connected to the route complex 102, a downstream PCI-PCI bridge 1103 connected to the peripheral device 109, and a PCI Express switch internal bus 1102 which connects the upstream PCI-PCI bridge 1101 and the downstream PCI-PCI bridge 1103 to each other.
A TLP input from the upstream PCI-PCI bridge 1101 or the downstream PCI-PCI bridge 1103 is transmitted to the downstream PCI-PCI bridge 1103 or the upstream PCI-PCI bridge 1101 connected to a destination of the TLP through the PCI Express switch internal bus 1102.
Referring to FIG. 3A, the upstream PCI-PCI bridge 1101 includes a PCI Express adaptor 201 which terminates a link of a PCI Express bus for connecting the PCI Express adaptor 201 and the route complex 102 and which exchanges a TLP with a TLP transfer logic 205, the TLP transfer logic 205 which transfers the TLP to a destination of the TLP, an upstream PCI-PCI bridge control logic 1201 which performs a process designated by the TLP addressed to the bridge 1101 and setting of the bridge 1101, a PCI-PCI bridge configuration resister 207 which provides a PCI Express constitution space, and a PCI Express switch internal bus adapter 1202 which performs a process required to send the TLP to the destination in accordance with a mounting mode of the PCI Express switch internal bus 1102.
The PCI Express adapter 201 includes a PCI Express physical layer 202 which transmits and receives a signal by using a signal of a physical specification conforming to the standard of the PCI Express, a PCI Express data link layer 203 which performs re-sending control of a TLP, and a PCI Express transaction layer 204 which exchanges the TLP.
On the other hand, referring to FIG. 3B, the downstream PCI-PCI bridge 1103 is different from the upstream PCI-PCI bridge 1101 shown in FIG. 3A in that the downstream PCI-PCI bridge 1103 includes a downstream PCI-PCI bridge control logic 1203 in place of the upstream PCI-PCI bridge control logic 1201. This difference is to perform control, such as processes related to a hot plug and hot removal of the peripheral device 109 in the downstream PCI-PCI bridge 1103, which is different from control in the upstream PCI-PCI bridge 1101 in relation to a process designated by a TLP addressed to the bridge 1103.
A PCI Express switch network shown in FIG. 1, an arbitrary one of a plurality of peripheral devices 109 can be connected to the CPU 101. However, since the network forms a closed hierarchical structure, the peripheral device 109 cannot be shared by a plurality of CPUs 101.
As a conventional method of solving this problem, an advanced switching interconnect (ASI) which distributedly connects a plurality of CPUs 101 and a plurality of peripheral devices 109 to a network to dynamically set connection between the CPUs 101 and the peripheral devices 109 is standardized. An example of the ASI is described in Protocol Interface #8 (PI-8) R1.0, ASI-SIG, February 2004, pp. 7-11.
Referring to FIG. 4, an ASI network 1301 includes a route complex side PCI Express-ASI bridge 1302 which is connected to the route complex 102 and has a function of encapsulating a TLP in an ASI packet to transmit and receive the ASI packet, an ASI switch 1303 which transfers an ASI packet to a port to which a destination of the ASI packet obtained by encapsulating the TLP is connected, a peripheral device side PCI Express-ASI bridge 1305 which has a function of encapsulating the TLP in an ASI packet to transmit and receive the ASI packet, and a fabric manager 1304 which manages connection between the route complex side PCI Express-ASI bridge 1302 and the peripheral device side PCI Express-ASI bridge 1305.
In this case, each route complex side PCI Express-ASI bridge 1302 is constituted by a PCI Express switch 1401 and an ASI network adapter 1403, and the peripheral device side PCI Express-ASI bridge 1305 is constituted by the ASI network adapter 1403 and the PCI Express switch 1601.
Referring to FIG. 5, the PCI Express switch 1401 of the route complex side PCI Express-ASI bridge 1302 is different from the PCI Express switch 801 shown in FIG. 2 in that the route complex side PCI Express-ASI bridge 1302 includes a downstream PCI-PCI bridge 1402 in place of the downstream PCI-PCI bridge 1103. The downstream PCI-PCI bridge 1402, as shown in FIG. 6, corresponds to a bridge obtained by removing the PCI Express adapter 201 from the downstream PCI-PCI bridge 1103. The downstream PCI-PCI bridge 1402 is directly connected by an internal bus because the PCI Express switch 1401 and the ASI network adapter 1403 are mounted in the same chip. The ASI network adapter 1403 has a function of encapsulating a TLP by using an ASI packet determined for each port of the PCI Express switch 1401 to transmit and receive the ASI packet.
On the other hand, referring to FIG. 7, the PCI Express switch 1601 of the peripheral device side PCI Express-ASI bridge 1305 is constituted by an upstream PCI-PCI bridge 1602 connected to the ASI network adapter 1403 and the downstream PCI-PCI bridge 1103 connected to the upstream PCI-PCI bridge 1602. The downstream PCI-PCI bridge 1103 has a configuration shown in FIG. 3B. In contrast to this, the upstream PCI-PCI bridge 1602, as shown in FIG. 8, corresponds to a bridge obtained by removing the PCI Express adapter 201 from the upstream PCI-PCI bridge 1101 shown in FIG. 3A. This is because, as in the downstream PCI-PCI bridge 1402 shown in FIG. 5, the ASI network adapter 1403 and the upstream PCI-PCI bridge 1602 are directly connected to each other by an internal bus. In FIG. 7, although one downstream PCI-PCI bridge 1103 is used for descriptive convenience, the number of downstream PCI-PCI bridges 1103 is not limited to one.
The fabric manager 1304 sets the ASI network adapters 1403 of both the bridges 1302 and 1305 to encapsulate a TLP in an ASI packet to perform tunneling between the downstream PCI-PCI bridge 1402 in the PCI Express switch 1401 of the route complex side PCI Express-ASI bridge 1302 and the upstream PCI-PCI bridge 1602 in the PCI Express switch 1601 of the peripheral device side PCI Express-ASI bridge 1305. This setting is performed by using a control ASI packet. In this case, by an application program operated on the CPU 101 or a request from an input/output interface, connections between the downstream PCI-PCI bridge 1402 and the upstream PCI-PCI bridge 1602 are changed as needed. With this operation, the plurality of peripheral devices 109 are shared by the plurality of CPUs 101.
As described above, by using the ASI network 1301 shown in FIG. 4, the peripheral devices can be shared by the plurality of CPUs. Furthermore, the ASI network is made on the assumption that the PCI Express switch shown in FIG. 2 is utilized. Even though a connection is performed to make it possible that three peripheral devices are shared by two CPUs, two PCI Express switches 1401, in each of which the total number of bridges is four and three PCI Express switches 1601, in each of which the total number of bridges is two, i.e., a total of five PCI Express switches are required. As a result, the total number of bridges is 14. In general, a connection is performed to make it possible to share m peripheral devices by n CPUs, a total of n(1+m)+2 m bridges are required. For this reason, in order to construct a system in which CPUs and peripheral devices are distributedly connected to a network, a circuit for bridges to connect the CPUs and the peripheral devices to the network disadvantageously increase in scale.